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Abstract 

This article explores the challenges that students face in navigating 
the curricular structure of post-secondary degree programs, and how 
predictive analytics and choice architecture can play a role. It examines 
Degree Compass, a course recommendation system that successfully 
pairs current students with the courses that best fit their talents and 
program of study for upcoming semesters. Data are presented to 
demonstrate the impact that this system has had on student success. In 
particular the data will show that by closing the information gap, this 
system is able to close the educational achievement gap for low-income 

and minority students. 
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Jit has been a longstanding reality that success in higher education is very uneven 
across the population of the United States. Consistently over the last three decades racial 
minority, low-income, and first generation students have earned post-secondary degrees 
at substantially lower rates than their counterparts. Although the degree-attainment rates 
for these three groups have increased over that time horizon, those improvements have not 
kept pace with the degree attainment rates of students in general (NASH & The Educational 
Trust, 2009; NCES, 2012; U.S. Census Bureau). The most recent IPEDS data show that whilst 
49 percent of white students who began college in 2007 graduated with at least an associates 
degree in 6 years, 37 percent of their African American counterparts, and 33 percent of 
Hispanic students graduated. While the rate at which low-income students enroll in higher 
education has doubled since the 1970s the graduation rate for these students has only 
grown from 7 percent to 10 percent (NASH & The Educational Trust, 2009; Postsecondary 
Education Opportunity. 1 ) First generation students begin to trail their peers as early as their 
first year, earning 18 credits, on average, compared to the 25 credits earned by students 
whose parents have degrees (Chen & Carroll, 2005). In fact, similar patterns emerge for 
minority, low-income, and first generation students in every success metric governing 
student progress through college when compared with their white, higher-income or non- 
first generation peers (Kelly, 2005; Lumina Foundation, 2014; NASH & The Educational 
Trust, 2009). 

These attainment gaps appear to be significantly influenced by information gaps. 
First generation, low-income and minority students often do not have the advice system 
that surrounds students whose parents or other relatives have been to college. Information 
is certainly available to these students, but without knowledge of the structure and 
nomenclature of higher education they are unable to even frame the questions that would 
enable them to become informed (Diamond et al., 2014; Hagelskamp, Schleifer, & DiStasi, 
2013; Kadlec, Immerwahr, & Gupta, 2014). 
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The process of navigating institutions from admission to graduation involves large 
numbers of crucial decisions, and once again, the information gap plays its part in the 
achievement gap. Despite the advantages to having a clear direction of study (Jenkins & 
Gho, 2012), one third of first generation students begin college without identifying a major or 
program of study, whereas only 13 percent of their peers with college-going parents do so (Chen 
& Carroll, 2005). Students select their majors with little information about what is involved 
in successfully completing the program, and often discover too late that the picture they had 
of that discipline is very different from the reality (Kirst & Venezia, 2004; Smith & Wertlieb, 
2005). Low-income and minority students express less knowledge of programmatic demands 
than their peers. Although students may think that they have an interest in a particular area, 
they receive little information about whether their academic abilities create a realistic chance 
of successfully completing that program. What is more, they may associate each discipline 
with a limited number of careers, and often eliminate disciplines from their list of choices 
because those jobs are unappealing, without realizing the true variety of career opportunities 
that lie on the other side of graduation. 

As challenging as the factors involved in choosing the right degree program are, 
navigating a degree program is no less crucial or challenging. Each student must choose from 
a variety of courses that satisfy the requirements of their general education core, and then 
their various degree program requirements. Ideally students would make strategic decisions 
about which courses are most likely to lead to their success. Instead, they are faced with 
making choices between courses that, ahead of time, they are not in a position to distinguish 
between. Indeed higher education has been described as a “post-experience good” (Diamond 
et al., 2014), since not only is it difficult to envisage or evaluate the experience of studying a 
particular course or program before hand, the true benefits of that study may not be understood 
until long into the future. Advisors are often well equipped to provide valuable advice in their 
own field. But, most programs require students to take courses from across the full spectrum 
of disciplines, and advisors find themselves challenged to offer useful advice in disciplines far 
from their own. As higher education funding has become more and more depleted, even access 
to this advice is far from guaranteed (Kadlec et al., 2014). 

Yet access to advising is vital as nationwide, college students take up to 20 percent 
more courses than are needed for graduation on average - not motivated by a desire for 
a diverse curriculum, but because they had to rethink their plans several times. In an 
environment in which time to degree has considerable implications for a student’s likelihood 
of successfully graduating, a semester of extra coursework plays a crucial factor, especially 
for students who attend part time, or for whom financial impacts weigh heavily (Complete 
College America, 2011). 

Information and choice clearly have a significant impact on a student’s ability to 
navigate through a degree successfully. But this significantly raises the stakes on the ways 
in which the information is presented and how the choices are framed. Schwartz (2004) 
has argued for a paradox of choice - that having too many options can lead to a decision 
paralysis. Tversky and Kahneman have carefully analyzed how decisions are made in the 
face of an abundance of choice (Kahneman, 2011; Kahneman & Tversky, 1979; Tversky & 
Kahneman, 1974). They, and others, have found that when presented with too many choices 
people fall back on a variety of rules-of-thumb, anecdotal evidence, or rely on cognitive 
ease and the halo effect. Often, poorer choices are made in situations of an abundance of 
choice, using these fall back methods, than in situations with more limited choice. In fact 
the literature on choice overload suggests that too many options can result in several adverse 
experiences including a depletion of cognitive resources and post-decision feelings of regret 
(Reed, DiGennaro Reed, Chok, & Brozyna, 2011; Schwartz, 2004). Given the multiplicity of 
choices entailed in selecting from a college’s array of majors or programs, and then satisfying 
the curricular requirements they require, these adverse experiences may play a significant 
part in student success, especially for at-risk populations. In fact it seems that a more focused 
choice structure would be far more effective and preferred (Diamond et al., 2014; Kadlec et 
al., 2014; Reed et al., 2011; Schwartz, 2004). 

While these educational achievement gaps have remained stubbornly present, one 
promising avenue of attack seems to be the use of predictive analytics to provide individualized 
information to each student, and so to more evenly level the information playing field. 
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Predictive analytic techniques move from a retrospective reporting data stance toward the use 
of large data sets to make detailed predictions about the future. These predictive models enable 
strategic action to be taken in the present to potentially provide significant improvements in 
the future. In this vein an appropriately designed system could use the perspective of the past 
to better inform students, and conversations between students and advisors. Such a system 
could allow advisors and students to make plans for future semesters, illuminated by the 
knowledge of courses or even majors in which past students with similar programs, grades 
and course histories had found success. It could also provide a focused choice architecture in 
which students could choose from a more limited selection of majors or courses that have been 
individualized to them, whilst leaving all possibilities available. 

Recent Work to Respond to this Challenge 

My recent work at Austin Peay State University and now at the Tennessee Board of 
Regents has, in part, been focused on finding ways to empower student choices by creating 
choice architectures that improve the information available to each student. The concept was 
to combine predictive analytics with behavioral economics to create an environment that 
would help students and advisors select impactful courses. We were intentional in providing 
an interface that neither restricts nor prescribes their choices, but instead empowers choice 
by creating an information source with a larger than human viewpoint and supported by data 
from previous choice patterns (Denley, 2012). 

Recommendation systems implemented by companies such as Netflix, Amazon and 
Pandora are a familiar feature of life today. We decided to create an interface in that vein, 
and developed a course recommendation system (Degree Compass) that successfully pairs 
current students with the courses that best fit their talents and program of study for upcoming 
semesters. The model combines hundreds of thousands of past students’ grades with each 
particular student’s transcript to make individualized recommendations for each student. 
However, the recommendations in this system had to be made within the confines of each 
student’s degree structure, and in a fashion that aligned more closely to the concerns of effective 
advising if it truly were to level the information field. In contrast to systems that recommend 
movies or books, these recommendations do not depend on which classes students like more 
than others. Instead it uses predictive analytics techniques based on grade and enrollment data 
to rank courses according to factors that measure how well each course might help the student 
progress through their program. In their 2009 book, Thaler and Sunstein discuss strategies 
to better structure and inform complex choices (Macfadyen, Dawson, Pardo, & Gasevic, 
2014). Degree Compass was designed with this in mind to create a choice architecture to 
nudge students toward course selections in which the data suggest they would have the most 
productive success, but using an interface that would minimize choice overload. 
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2 Degree Compass is now a commercially marketed product, available from D2L Incorporated 
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The algorithm liaises with the institution’s degree audit system to find the courses 
that would satisfy some as yet unsatisfied degree requirement, if the student were to take that 
course. From these courses that could apply directly to the student’s program of study, the 
system selects those courses that best fit the sequence of courses in their degree, recommending 
courses that are curricularly more central before those which are more specialized. That 
ranking is then overlaid with a model that predicts the courses in which the student will 
achieve their best grades. In this way, the system most strongly recommends those courses 
which are necessary for a student to graduate, core to the institution’s curriculum and their 
major, and in which the student is expected to succeed academically. 

The recommended course list is conveniently displayed in a web-based interface 
on the secure side of the institution’s information portal. This interactive interface provides 
information on each recommended course’s curriculum and requirements, what role that 
course plays in the student’s degree, as well as class availability in upcoming semesters. The 
student is able to filter the list to show only classes that are offered online, or face-to-face, or 
only at particular campuses to refine their decisions according to some practical constraints. 


The concept was to 
combine predictive 
analytics with behavioral 
economics to create an 
environment that would 
help students and advisors 
select impactful courses. 


The strength to which the system recommends each particular class is communicated 
by a star rating. A five star class is one that, amongst the presently available courses, best fits 
the student’s curricular constraints, and is one in which the student is predicted to earn as 
good a grade as they might earn in any other course that would fulfill their requirements. It 
does not necessarily mean that they will get an A grade. Indeed the interface does not reveal 
predicted grades to the student. However, all of this information is available to advisors as a 
tool for academic advising that supplements the information available when providing advice 
to their advisees. 


The interface also provides a majors recommendation system called My Future. 
For a student who has already identified their major, My Future provides information about 
concentration choices and degree pathways, as well as links to prospective career paths, job 
availability and 0*Net statistics for graduates in that major. For a student who is yet to 
choose a major, or is thinking about changing their major, it provides a list of majors in 
which that student is predicted to be the most academically successful. Again, for each of 
these majors, information is provided about concentration choices and degree pathways as 
well as prospective career paths and job availability. My Future uses data-mining techniques 
to identify the courses that are the best indicators of success in each of the institution's 
programs - the courses that capture the flavor of each major - and uses Degree Compass’ 
technology to predict course grades and find the majors in which each student will be the 
most academically successful. 
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The system was developed in collaboration with faculty, advisor and student input to create 
an interface that would be able to supplement the advising process. The interface itself was 
developed to allow commonly utilized functionality in a familiar format. When developing 
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the grade prediction engine for these tools, we chose the data sources on which to base the 
predictions carefully. Since one of the objectives was to try to impact the performance of 
subpopulations for which there has been an achievement gap in the past, we chose not to 
use any demographic information in the model. We also chose to make the system faculty- 
agnostic by not disaggregating the grading patterns of different faculty. Conversations with 
faculty members suggested that by doing this there would be greater faculty involvement in the 
project, and greater utility for the tool. 

What the Data Say about the Impact of Degree Compass 

We developed a strong assessment structure to assess the impact of Degree Compass 
on student success (Denley, 2013). Data collected as part of the Degree Compass project fell 
largely into three categories. First, because courses are recommended to students based on 
curricular fit, together with a prediction of the grade that student would earn if they were to 
take the class, it is crucial to collect data that establish the accuracy of the grade predictions. 
Degree Compass was built to track the predicted grade as well as the earned grade for each 
student in each semester in each class in which they were enrolled. Secondly, given that advice 
from Degree Compass is useful only if it is consulted, the system used click-traffic data to 
provide information about the system’s use. Focus groups and surveys also provided feedback 
about the usability of the interface and other features that users might consider informative. 
Finally, the aim of the project was to empower students to make more advantageous choices in 
their education that would help them move effectively through their curriculum. Consequently 
we measured student success and progression through their curricula. 

Our initial results for the 10,873 students at Austin Peay State University (APSU) 
were very encouraging. However, it was important to establish that our modeling techniques 
could calibrate themselves to differing institutional settings and student populations. Generous 
support from Complete College America and the Bill and Melinda Gates Foundation allowed 
us to replicate the system at three other schools in Tennessee - two community colleges and 
one university - adding another almost 40,000 students. Fortunately, the results from all three 
campuses replicated the ongoing grade prediction resolution achieved at APSU. Data from Fall 
2012 showed that the average predicted grades in the university settings were within 0.59 of 
a letter grade of the awarded grades, and 89 percent of those who were predicted to pass the 
course indeed passed. In the community college setting, average predicted grades were within 
0.64 of the awarded grades, and 90 percent of students who were predicted to pass the course 
did so. These results confirmed that the grade prediction engine successfully predicts grades 
in settings across the higher education spectrum, from a rural community college to an urban 
research university. 

Of course, the motivation behind this work was not to predict grades, but rather 
to provide a choice architecture in which students and advisors could make more nuanced 
decisions about degree programs. Using Degree Compass as part of academic advising at APSU 
has steered students towards more classes in which they would more readily succeed, both 
by passing the course in greater numbers and also achieving higher grades. A comparison 
of student grades before the introduction of the system with those today shows a steadily 
increasing ABC%, with grade results across the institution today more than 5 standard 
deviations better than those in Fall 2010. This very statistically significant shift was apparent 
across the student body, from freshmen to seniors. We saw similarly significant increases for 
several subpopulations, including African American students (an increase of 2.1 percent, with 
2.89 standard deviations) and Pell recipients (an increase of 3.9 percent, with 7.7 standard 
deviations). These figures are not results from a sampling of the student population, but include 
the entire undergraduate student body. 

While it is still early to make general connections between Degree Compass and 
graduation rates, since the system was introduced at APSU in Spring 2011, the six-year 
graduation rate has increased from 33 percent to 37.4 percent, with the greater gains for low- 
income students (increased from 25 percent to 31 percent) and African American students 
(increased from 28.7 percent to 33.8 percent). 

On a more granular level we carried out a detailed analysis of the data to connect 
Degree Compass recommendations with student successes in their classes and progression 
through their degrees. Historically, the grade distributions across all four campuses, of all 
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students, showed a picture in which 63 percent of the time a student received an A or a 
B grade in their course. Using Degree Compass, a much larger proportion of the students 
who were predicted to earn a B or above were actually awarded that grade. Indeed, on each 
campus more than 90 percent of students who took a course in which they were predicted 
to get at least a B actually earned an A or a B grade. The analysis shows that this effect was 
evidenced at every school and at every course level from developmental classes through 
upper-division courses. 

For each of the institutions the number of earned credits was highly correlated 
with number of recommended classes that were part of a student’s semester schedule. For 
instance, those students who took a 12-hour schedule that contained no recommended 
classes earned only 2.5 credits on average, compared with 10.5 credits for those students 
whose entire schedule was crafted from recommended courses (see Figure 1). Analysis of other 
attempted loads showed similar results. With correlation coefficients ranging from 0.7 to 0.9, 
this connection translates into significant gains when students take recommended classes in 
comparison with taking classes that are not recommended. 



Number of recommended courses in semester schedule 


Figure 1. Comparison of average earned hours in a 12-hour schedule disaggregated by the number of 
recommended classes. 


Further analysis of attempted and earned hours revealed that the achievement gap 
between the average hours earned by white students and average hours earned by African 
American students reduced significantly for those students who took classes recommended by 
Degree Compass. For instance among students who attempted 12 hours, white students earned 
10.06 hours on average, while their African American peers earned 8.06 hours on average. As 
we have seen, this is the familiar achievement picture nationally. However, for those students 
who took 12 hours of courses all of which were recommended by Degree Compass, all students 
did better, regardless of ethnicity. White students earned 11 hours while African American 
students earned 10.3 hours on average. The 20 percent achievement gap was more than cut in 
half (see Figure 2). We see much the same picture for low-income students. Among students 
who attempted 12 hours, low-income students earned 8.35 hours on average, while their peers 
earned 10.07 hours on average. However, for those students who took 12 hours of courses, all 
of which were recommended by Degree Compass, low-income students earned 10.3 hours 
while their peers earned 11.04 hours on average. Once again, all students did better, and again 
the achievement gap was cut in half. 

Conclusion 
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Degree Compass has crystalized a number of topics concerning the role that predictive 
analytics might play in higher education and student success initiatives in particular. First, as a 
proof of concept, it is now apparent that student success interventions powered by predictive 
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Average Gap 
for All Schedules 



Students 


Average Gap 
with Degree Compass 



White Minority 
Students 


Figure 2. Comparison of average earned hours from a 12-hour schedule disaggregated by race for 
students in general and students who took only courses recommended by Degree Compass. 

analytics are capable of moving the needle on degree completion. The performance data above 
clearly demonstrate that students in both the university and community college settings 
progress more effectively through their degree programs when they follow a course sequence 
informed by data-analytics. Furthermore, there have been precious few approaches that have 
been able to appreciably close the educational achievement gaps for race and income, and 
fewer still that can be scaled. Once again, the data suggest that this approach is one that is 
effective and can be broadly applied at scale. 

This approach, however, has highlighted a number of educational issues. It is clear 
that in a model that uses the past to influence the future there is the danger of perpetuating or 
even reinforcing existing stereotypical trends. However this need not be the case. One of the 
reasons we chose not to employ demographic information as part of the predictive modeling 
was precisely to build in safeguards against such phenomena. The system is designed to be 
able to use additional data sources as they become available. However, the data that we have 
collected so far seem to suggest that our current approach has been successful. 

This is not computer¬ 
ized decision making, 
but technology-informed 
choice. 


One of the important features of Degree Compass is that it only suggests courses 
that satisfy existing degree requirements. The curriculum is only as rigorous as the courses 
that can be taken to navigate it, and those remain unchanged. Consequently, the courses 
that are suggested by the technology are courses that any student might always have chosen 
and any advisor might always have advised a student to take. The issue comes down to how 
a student’s or advisor’s knowledge of the curriculum might inform that choice. It is also an 
important observation that the suggestions are just that. This is not computerized decision 
making, but technology-informed choice. The software provides additional information which 
the student and advisor are then able to use to make more informed decisions. The influence 
of a plausible default is an important aspect of this, and is an intentional feature of the choice 
architecture provided in the interface, but the choices that the student and advisor make are 
still their free choice. 


In a similar vein, by nudging students towards courses in which they are predicted to 
have greater success there is the possibility that we may erode academic rigor by systematically 
steering students towards the easy classes. It may be interesting to contemplate whether when 
a student takes a class in which they have an increased likelihood of success they are taking 
an easier class. The experience in the class is as much a function of the student’s preparation 
or talent as it is the challenge of the course. Indeed, as faculty we are all guilty of following the 
easier route and studying a topic in which we had talent and insight rather than taking the 
academically more challenging route of choosing a subject for which we had no affinity. 
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The system only ever suggests courses that satisfy unmet degree requirements. This 
has the potential to reduce the numbers of excess hours that students currently take. By only 
suggesting courses that meet degree requirements there is the possibility that the students’ 
experience of the aspect of discovery and intellectual curiosity in the educational process may 
be stifled. However, transcript analysis shows that more often than students choosing courses 
off their curricular path because of intellectual curiosity, they actually take these classes 
simply because the course they would like to choose is unavailable. Since the data now clearly 
support that students taking the courses that they need is a crucial aspect of student success, it 
is incumbent on us to offer the classes that students need, when they need them. If we employ 
predictive technology to ensure that the skeletal structure of the degree is seamlessly available 
to students, we create the flexibility for more intellectual curiosity should the student choose. 

In fact, a deep dive into 
data at the Tennessee 
Board of Regents has 
allowed me to create 
strategic insights into the 
structure of the system 
and how students succeed 
and fail. These insights 
are being used to inform 
changes to system policy, 
as well as direct broad- 
scale system initiatives. 


Here we have concentrated on seeing how individualized analytics can be used to help 
optimize course and curricular selections, but there are many other ways in which these 
kinds of technology can be utilized across higher education. This work demonstrates how 
predictive analytics can provide a larger-than-human viewpoint that can inform student 
choice. We are starting to see how these kinds of recommending systems can empower 
decisions by program coordinators, and institutional leadership. In fact a deep dive into 
data at the Tennessee Board of Regents has allowed me to create strategic insights into 
the structure of the system and how students succeed and fail. These insights are being 
used to inform changes to system policy, as well as direct broad-scale system initiatives. It 
seems likely that over the coming years we will see more and more ways in which predictive 
analytics and data-mining technology coupled with behavioral economics will play roles in 
higher education on every scale (Johnson et al., 2013; cf. O'Reilly & Veeramachaneni, 2014). 
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